Skip to content

Gateway waves 1–7: integrate the gap-analysis features - #14

Merged
ALikesToCode merged 206 commits into
mainfrom
integrate/gateway-wave2
Oct 9, 2026
Merged

ALikesToCode merged 206 commits into
mainfrom
integrate/gateway-wave2

Conversation

@ALikesToCode

@ALikesToCode ALikesToCode commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Now carries Waves 1–7

The integration stack (#25, #39, #53, #60, #69, #76) was merged into this branch from the top down, so merging this PR lands all seven waves (#7–#76) and the dependency-audit fix (#42) on main. This branch's tree is identical to integrate/gateway-waves-1-7 at 27c5f1d, which passed every CI job.

D1 migrations 0014 through 0038 were applied to multillm-intelligence before this merge. Every new feature is off by default. The sections below describe Wave 1 as it was first opened.

Combines Wave 1 of the gateway gap analysis into one branch, so the features can be reviewed and merged together.

Included

PR Feature Default
#7 Gap analysis of 1,185 gateway repositories docs only
#8 Upstream outcomes: circuit recovery separates valid responses, credential rejection, throttling and ambiguous transport failure always on, no config
#9 First-event stream validation for auto: Chat streams MULTILLM_STREAM_PREFLIGHT=off
#10 Provider authenticity and conformance diagnostics (operator script) run by hand
#11 Candidate-specific roleplay fallback context ROLEPLAY_CANDIDATE_CONTEXT_REFIT=off
#12 Native provider token counts for /v1/messages/count_tokens estimate mode, NATIVE_TOKEN_COUNT_ENDPOINTS_JSON={}
#13 Prometheus scrapes: public route health and admin request window PROMETHEUS_ENABLED=false

Merging this PR merges all seven, and closes them as merged once GitHub sees their commits on main.

Conflict resolution

Verification on the merged tree (clean worktree, no .env)

  • Python: 2779 passed, 2 skipped. ruff, mypy and bandit as in CI: clean. test_proxy_documentation: 21 passed.
  • npm run test:worker: 1204/1204.
  • test:deploy 13/13, free routes 8/8, client headers 12/12.
  • cf:migrations:check passes; no new migrations.
  • wrangler deploy --dry-run bundles: 786.64 KiB, 174.45 KiB gzip.
  • tests/test_frontend_build.mjs fails locally on main too, because @tailwindcss/cli isn't installed here.

Deploy note

Merging to main deploys through Workers Builds. With the defaults above, live behaviour is unchanged apart from #8's outcome classification. No D1 migration is needed.

Compares MultiLLM against the 1,185 repositories tagged llm-gateway on GitHub, ranks the 40 largest gaps with evidence on both sides, and specifies the first six features to build.
Provider circuits now classify each transport result into independent provider-health, credential-health and replay signals. Credential rejection, throttling and caller errors no longer heal a degraded or open circuit, and they release half-open probes without an outage penalty. Retry rules and raw single-forward behaviour are unchanged.
Adds MULTILLM_STREAM_PREFLIGHT (off by default). In strict mode, streaming auto-route Chat requests are held until the first useful text, reasoning or tool-argument delta arrives within a bounded prefix, then the buffered bytes are replayed once in order. Empty, malformed or errored prefixes return a real 502 or 504 and never trigger another upstream request.
Adds scripts/provider_authenticity.py, an operator tool that checks one provider-qualified model through the gateway. By default it sends one metadata request and no generation. With --allow-generation it runs at most two 64-token nonce probes and reports narrow supported, contradicted or unknown verdicts without raw prompts, completions or credentials.
Adds ROLEPLAY_CANDIDATE_CONTEXT_REFIT (off by default). In window mode each fallback candidate gets a context view that fits its own window: the oldest complete unprotected message groups are omitted while directives, the latest user turn and tool-call pairs stay intact. Payloads, prompt-cache annotations, continuation and headers use the selected view; stored history and recovery keep the full conversation.
The Worker now passes MULTILLM_STREAM_PREFLIGHT through its Container environment allowlist, so the setting works on the Cloudflare deployment as well as standalone Flask. The README lists the feature and the guide describes deployment.
The guide now states that the classification needs no configuration.
The guide now describes how the candidate context tests run.
The README feature list links the diagnostics guide.
POST /v1/messages/count_tokens accepts X-MultiLLM-Token-Count-Mode. estimate stays the default and keeps its contract. auto uses a configured provider count and falls back to a labeled estimate. native requires a provider count and returns 501 when none is configured.

NATIVE_TOKEN_COUNT_ENDPOINTS_JSON maps a registered provider to its verified count path and defaults to {}. The adapter sends one bounded, count-only request with redirects and retries disabled. Upstream errors keep their status in an Anthropic error envelope. Only native counts reach a provider, so only they check the key's model allowlist.
PROMETHEUS_ENABLED=true serves two Prometheus 0.0.4 text endpoints and defaults to off, where both return 404.

/status.prometheus is public. The Worker renders it from the D1 status snapshot, so a scrape never wakes the Container, and it exposes only the fixed public route, provider and overall states.

/v1/metrics/prometheus requires an admin user with an admin-scoped API key. Flask renders it with aggregate 24-hour request counts by response class plus p50/p95 latency.

Both endpoints cap output at 512 series and 256 KiB and report truncation in a gauge.
…egrate/gateway-wave2

# Conflicts:
#	package.json
#	tests/fixtures/secret_firewall_coverage.json
@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8508eb5a-f126-4d0d-9ae6-e6319602fa12

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T19:46:23.688361Z e0c6b66 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e0c6b6644f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread worker/roleplay/endpoint.mjs Outdated
state,
candidate,
messages: roleplayMessages,
messages: selectedMessages,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve per-candidate windows during refusal fallback

When a large-window GLM candidate returns the semantic-refusal sentinel and the configured uncensored fallback has a smaller context window, passing only selectedMessages here loses the fallback candidate's previously computed contextPlan. createRoleplayContinuation recalculates capacity from the first candidate's larger view without refitting and can report roleplay_continuation_context_exhausted, returning the generic fallback-failed response without ever dispatching the otherwise eligible smaller candidate. Carry the per-candidate view into semantic-refusal dispatch rather than reusing the selected candidate's messages for every fallback.

Useful? React with 👍 / 👎.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
🔵 In progress
View logs
multillm-proxy f33ffa7 Oct 09 2026, 10:41 AM

Provider catalogs can declare a model's precision (fp32, fp16, bf16, fp8, int8, int4) and quantization scheme. MODEL_PRECISION_PREFERENCE takes an ordered JSON list of precisions and is unset by default. While unset, catalog output and auto:intelligence candidate order are unchanged.

When set, precision is recorded only from a provider's explicit catalog fields, never inferred from a model name. It breaks ties between candidates the quality and fast profiles already rank equally. It never adds a candidate, crosses a ranking group, reorders the balanced chain or changes an explicitly requested model. The intelligence policy can override the setting with its own precision_preference list. An invalid environment value is logged once and treated as unset.
RESPONSE_CACHE_POLICY_REVISION selects the response cache identity. legacy, the default (also when unset or empty), keeps today's keys, bodies and headers. v1 adds a canonical digest of the resolved route, the key's model permissions, its secret-scan mode, the retention policy and the workflow version. A policy change therefore misses instead of replaying an answer produced under different rules.

Under v1, routes whose effective plan is not fully known (free pools, cascades, health-ordered auto routes, query parameters or provider-selection headers) bypass the cache. An answer is not stored if the policy changed while it was being generated. Unknown revisions bypass the cache, and old entries are never purged, only left unreachable.
scripts/probe_cost_forecast.py reads a JSON plan (from a file or stdin) describing the probe schedule, routes and prices. It prints upper bounds on provider calls and USD per day, per month and per run, and exits nonzero when a supplied --max-calls or --max-cost-usd limit would be exceeded.

The script never runs probes or contacts providers, and does not read the application environment. A price the plan does not state is reported as null, never as zero.
GET /openapi.json returns an OpenAPI 3.1 description of the public client API: Chat Completions, Responses, Anthropic Messages and token counting, models and image generation, including the opencode and codex-easy paths. It also covers the documented MultiLLM headers and the error and SSE envelopes. It requires a dashboard login or an API key with the models scope.

services/openapi_spec.py builds the spec without reading provider configuration, and docs/openapi.json is its committed output. A test fails when the two drift. Administrator routes are excluded.
Three tests started this machine's virtualenv Python by absolute path, so
they failed on GitHub Actions once the dependency audit stopped masking
them. The Worker tests now use PYTHON, then .venv, then python3, as the D1
integration test does, and the node job installs the locked Python
dependencies.
On a slow runner the 20 ms deadline expired before the request reached the
fetcher, so the hold was released instead of kept as unknown. The fetcher
never settles, so a 1000 ms deadline still fires after handoff.
The test checks admission uses the 1000 ms routing deadline rather than the
5000 ms policy one; a 3000 ms bound still tells them apart.
… test

The request takes about 80 ms, but a loaded GitHub runner took 530 ms. The
call count assertion already rules out a second attempt.
…ails

An intermittent CI failure returned 502 with no log, so the assertion now
reports the provider call count and the response body.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

feat(gateway): wire the Wave 7 enterprise features together
feat(gateway): integrate the Wave 6 gateway features
feat(gateway): integrate the Wave 5 gateway features
feat(gateway): integrate the Wave 4 gateway features
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@gitguardian

gitguardian Bot commented Oct 9, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
37552414 Triggered Generic Password c8787fc worker/alert-delivery.mjs View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@ALikesToCode ALikesToCode changed the title Gateway wave 1: integrate seven gap-analysis features Gateway waves 1–7: integrate the gap-analysis features Oct 9, 2026
@ALikesToCode
ALikesToCode merged commit c6e9b7c into main Oct 9, 2026
5 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the integrate/gateway-wave2 branch October 26, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant